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^ (54) Title: METHOD FOR ANIMATING A 3-D MODEL OF A FACE 

^ (57) Abstract: A method for animating a 3-D model of a person's face is disclosed. The 3-D face model carries both the geometry 
^ (shape) and the texture (color characteristics of the person's face. The shape of the face model is represented via a 3-D triangular 
— mesh (geometry mesh), while the texture of the face model is represented via a 2-D composite image (texture image). A separate 
^ 3-D triangular mesh, called the shape mesh, is used to animate the nodes of the geometry mesh: Each triangle of the shape mesh 
controls the motion of a plurality of nodes of the geometry mesh that are connected to it. Thus, by moving the nodes of the shape 
Q mesh, which are small in number, the nodes of the geometry mesh, which can be very large in number, are animated realistically. 
£^ The nodes of the shape mesh can be moved directly or indirectly. In the indirect method, the nodes of the shape mesh are moved in 
^ accordance with the so-called facial action values, which are even smaller in number than the nodes of the shape mesh. 
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METHOD FOR ANIMATING A 3-D MODEL OF A FACE 

CROSS-REFERENCE TO RELATED APPLICATIONS 

This application claims the benefit of U.S. Patent Application Serial No. 
09/689,566, filed October 12, 2000 (Attorney Docket No. 89589.112900). 



10 FIELD OF THE INVENTION 

The present invention is related to the field of computer animation, and more 
specifically, to a technique for animating a personalized three-dimensional (3-D) model 
of a person's face in accordance with the facial motion of the person. 

V" 

15 BACKGROUND OF THE INVENTION 

The goal of facial animation is to mimic the facial motion of a person on a 3-D 
model of the person's face as accurately and as fast as possible. The accuracy of a facial 
animation method is measured by the realism of the animated face it generates. On the 
other hand, the computational speed of a facial animation method determines whether it 

20 can be realized in real-time. 

The 3-D model of a person's face is composed of a 3-D triangular mesh, referred 
to as the geometry mesh, and an associated composite image of the person's face, referred 
to as the texture image. A 3-D triangular mesh refers to a connected set of triangular 
patches in 3-D whose corners form the nodes of the mesh. Each triangular patch in the 

25 geometry mesh acquires its image data from an associated triangular region in the texture 
image. The geometry mesh represents the geometry of the person's face in its neutral 
state. Animating the 3-D face model of a person involves deforming the geometry mesh 
of the face model to reflect the changes in the geometry of the face caused by the motion 
of the face. 

30 The methods disclosed in the prior art on facial animation can be generally 

classified as (i) physics-based methods and (ii) rule-based methods. In physics-based 
methods, the motion of each triangle of the geometry mesh is controlled by a multi-layer 
facial muscle system. Dynamic models of the facial muscles are employed to calculate 
the propagation of any facial force throughout the face and to obtain the resulting 

35 deformation of the surface of the face. Physics-based methods can produce realistic 
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animations, however, because of their high computational cost, they cannot be used in 
real-time applications. 

In rule-based methods, a subset of the nodes of the geometry mesh, referred to as 
feature points, are used to control the movement of the rest of the nodes of the geometry 
5 mesh. Each feature point is assigned an area of influence on the geometry mesh. When a 
feature point is moved, the nodes of the geometry mesh that belong to the area of 
influence of the feature point move according to some predefined deformation rules. 
These deformation rules may specify linear, piece-wise linear, or rotational motion for the 
nodes of the mesh with the amount of motion being inversely proportional to the distance 
10 of the node to its controlling feature point. Although the rule-based methods provide real- 
time deformations of the face, they may lack realism as they are not based on any 
physical model. 

SUMMARY OF THE INVENTION 

15 The present invention provides an improvement designed to satisfy the 

aforementioned needs. Particularly, the present invention is directed to a computer 
program product for animating a 3-D face model realistically and in real-time by 
performing the steps of: (a) receiving the 3-D face model of a person; (b) receiving the 
global and local facial motion values; and (c) animating the fine geometry mesh of the 3- 

20 D face model using a sparse shape mesh overlaying the geometry mesh. 



BRIEF DESCRIPTION OF THE DRAWINGS 

In the course of the following detailed description, reference will be made to the 
attached drawings in which: 
25 FIG. 1 is a perspective view of a computer system for implementing the present 

invention; 

FIG. 2 is a diagram illustrating the method of present invention; 
FIG. 3 is a flowchart for the method of the present invention; 
FIG. 4 is a diagram illustrating the method of receiving the geometry data set; 
30 FIG. 5 is a diagram further illustrating the method of receiving the geometry data 

set; 

FIG. 6 is a diagram illustrating the action faces used for animation; 
FIG. 7 is a diagram illustrating the method of attaching the geometry mesh to the 
shape mesh. 

2 
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FIG. 8 is a diagram illustrating the method of animating the face using the shape 

mesh. 

DETAILED DESCRIPTION OF THE INVENTION 

5 Referring to FIG. 1, there is illustrated a computer system 10 for implementing 

the present invention. The computer system 10 includes a microprocessor-based unit 12 
for receiving and processing software programs and for performing other well known 
processing functions. The software programs are contained on a computer useable 
medium 14, typically a compact disk, and are input into the microprocessor-based unit 12 

10 via the compact disk player 16 electronically connected to the microprocessor-based unit 
12. As an alternate to using the compact disk 14, programs could also be contained in an 
Internet server 18 and input into the microprocessor-based unit 12 via an Internet 
connection 20. A camera 22 is electronically connected to the microprocessor-based unit 
12 to capture the 2-D images of a person's face. A display 24 is electronically connected 

15 to the microprocessor-based unit 12 for displaying the images and user related 
information associated with the software. A keyboard 26 is connected to the 
microprocessor based unit 12 for allowing a user to input information to the software. A 
mouse 28 is also connected to the microprocessor based unit 12 for selecting items on the 
display 24 or for entering 2-D position information to the software, as is well known in 

20 the art. As an alternate to using the mouse 28, a digital pen 30 and a digital pad 32 may be 
used for selecting items on the display 24 and entering position information to the 
software. The output of the computer system is either stored on a hard disk 34 connected 
to the microprocessor unit 12, or uploaded to the Internet server 18 via the Internet 
connection 20. Alternatively, the output of the computer system can be stored on another 

25 computer useable medium 14, typically a compact disk, via a compact disk writer 36. 

The below-described steps of the present invention are implemented on the 
computer system 10. Before describing the steps of the present invention, it will facilitate 
understanding to have an understanding of the following terms. Referring to FIG. 2, 3-D 
face model is composed of a 3-D triangular mesh (geometry mesh) 41 and a 2-D 

30 composite image (texture image) 42. 3-D triangular mesh refers to a connected set of 
triangular patches in 3-D whose comers form the nodes of the mesh. Each triangular 
patch 43 in the geometry mesh 41 is associated with a triangular region 44 in the texture 
image 42. In order to render the face model of a person on the display 24 of the computer 
system 10, the outside surface of each triangular patch 43 in the geometry mesh 41 is 
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painted with the image data contained in its corresponding triangle 44 in the texture 
image 42. Image data are transferred from a triangle 44 in the texture image 42 to its 
counterpart 43 in the geometry mesh 41 via an affine transform which is well known to 
anyone knowledgeable in the field of computer graphics. Although we have disclosed 
5 triangular meshes, those skilled in the art will understand that any other polygonal mesh 
could be substituted for the triangular mesh. 

Referring to FIG. 3, there are illustrated the five steps of the present invention 
which are first succinctly outlined and later described in detail. Briefly stated, the four 
steps are as follows: (a) receiving the 3-D face model of a person (Step 120); (b) 
10 receiving the global and local facial motion values (Step 130); (c) animating the 3-D face 
model using the shape mesh and the facial motion values (Step 140); (d) displaying the 
animated 3-D face model (Step 150). 

A. Receiving The 3-D Face Model (Step 120) 

15 

Referring to FIG. 3, the method of receiving the 3-D face model of the person 
comprises the steps of (1) receiving the geometry data set (Step 121); (2) receiving the 
texture image (Step 122); and (3) receiving any synthetic facial objects (Step 123). In the 
following, a detailed description of these steps is given 

20 

Al. Receiving The Geometry Data Set (Step 121) 

The geometry data set for a 3-D face model consists of an initial geometry mesh, a 
shape mesh, the positions of the nodes of the shape mesh for the neutral state and the 
25 action states of the face, and the attachment coefficients for the geometry mesh. An 
itemized description of the aforementioned components of the geometry data set is given 
in the following: 

1 . Initial geometry mesh: Referring to FIG. 4, the topology of the geometry mesh 
30 271 is received as part of the geometry data set. Hence, what is received as the 

first item is the number of nodes and triangles of the geometry mesh 271, and 
the information of how they are connected. 
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2. Shape meshes for the neutral and action faces: Referring to FIG. 5, the shape 
mesh 275 is a lower resolution (coarser) triangular mesh overlying the 
geometry mesh 271 and comprising substantially fewer and larger triangular 
patches than the geometry mesh 271. The shape mesh 275 is used for setting 
5 the coordinates of the nodes of the geometry mesh 271 for the neutral face. 

The shape mesh 275 is also used for animating the geometry mesh 271 in 
accordance with the motion values. Neutral face means that all face muscles 
are relaxed, eyes are normally open, mouth is closed and lips are in contact. 
Action faces refer to maximum motions of regions of the face such as during 
10 smiling or being surprised. 

Referring to FIG. 6, in a preferred embodiment of the invention there are a 
total of 5 action states for the face, namely, yawning-mouth 241, smiling- 
mouth 243, kissing-mouth 245, raised-eyebrows 247, and squeezed-eyebrows 
15 ' 249, in addition to the neutral state 221 of the face. In mathematical terms, let 

K denote the number of nodes of the shape mesh 275 and let D n , n - 1,..., K 9 
denote the positions of the nodes of the shape mesh 275 for the neutral face 
221. Let ifdenote the number of action faces and D nJ9 i = , denote the 

positions of the nodes of the shape mesh 275 for the action faces. Hence, what 
20 is received in the second step as part of the geometry data set is the topology 

of the shape mesh 275 and the coordinates of the nodes of the shape mesh for 
the neutral face D n , n = l 9 ... 9 K, and optionally for the action faces D n ii 
i = l,...,H. 

25 3. Attachment coefficients: Referring to FIGS. 4 and 7, the attachment 

coefficients are used to attach the geometry mesh 271 to the shape mesh 275 
to obtain the geometry mesh 273 adapted for the person. The attachment 
method is such that a node 280 of the geometry mesh 273 is attached to a 
triangle 281 of the shape mesh 275 if there is a point on the triangle 281 such 

30 that the line passing through the node 280 and the point is parallel to the 

surface normal vector n p at the point. Still referring to FIG. 7, the node 280 of 
the geometry mesh 273 located at Q is attached to the triangle 281 of the 
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shape mesh 275 because the line passing through Q and P is parallel to n p . 
Then, it is said that the node Q 280 is attached to the triangle ABC 281 at the 
point P . This attachment is quantified by five numbers, namely the weights 
a , p , y , the distance d between the node Q 280 and the attachment point 
5 P, and the index m of the triangle 281. Hence a, p y , d, and m are 

referred to as the attachment coefficients. The 3-D position of the node Q 280 
is expressed as 

10 

where A, B, and C denote the positions of the corners of the triangle 281, and 
n is obtained as a normalized weighted average of the normal vectors n A , n B 
and n c at the corners A, B, and C, respectively, of the shape triangle 281 as 
follows: 

1j n T\ ~ if • 

pn A +/3n B +yn c \\ 

A different set of attachment coefficients is received for each node of the 
geometry mesh 273 as part of the geometry data set. In mathematical terms, 
let L denote the number of nodes in the geometry mesh 271. Then, the 
20 attachment coefficients are denoted by a k> p t , y k , d k9 and m k for 

fc = l,...,Z,. Hence what is received in the third step is the attachment 
coefficients a k , fi k ,y k , d k , and m k for the nodes of the geometry mesh 273. 

A2. Receiving The Texture Image (Step 122) 

25 

The texture image is used to paint the surface of the geometry mesh of the person. 
The texture image may consist of a single image, a mosaic of multiple images, or simply 
a concatenation of multiple images. Associated with the texture image, affine 
transformations are also received for every triangle of the geometry mesh 273. The affine 
30 transformations are used to map the texture image data onto the triangles of the geometry 
mesh 273 as is well known to anyone in the field of computer graphics. 
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A3. Receiving Any Synthetic Facial Objects (Step 123) 

The face model may include synthetic objects such as eyeglasses and earrings. A 
5 3-D mesh model of a pair of eyeglasses is selected from a collection of eyeglasses as 
indicated by an eyeglasses identifier, if the face model includes eyeglasses. Similarly, a 3- 
D mesh model of a pair of earrings is selected from a collection of earrings as indicated 
by an earrings identifier, if the face model includes earrings. The 3-D meshes of the 
synthetic components are attached to the geometry mesh of the face to obtain the 
1 0 complete face model of the person. 

B. Receiving The Global And Local Facial Motion Values (Step 130) 

15 Referring to FIG. 3, in the third step of the invention, the global and local motion 

values for animating the 3-D face model are received in a chronological order. The global 
motion is represented by the 3-D rotation matrix R f and the 3-D position vector T f . The 
superscript / denotes the chronological order number for the motion values. 

20 The local motion is represented by the 3-D position values D{ , n = 1,..., JT, of the 

nodes of the shape mesh 275. The superscript/ denotes the chronological order number 
for the position values, and K denotes the number of nodes in the shape mesh 275. The 
position values D f n are either received directly or indirectly. In the indirect method, the 

action vectors A f are received instead, and the position values are calculated from the 
25 action vectors using the following matrix-vector equation: 

D' a =[D ntl -D n ... D ntH -D„]-A'+D n , 

where D n denotes the positions of the nodes of the shape mesh 275 for the neutral face 

30 221, and D n i for i = l,...,H , denote the positions of the nodes of the shape mesh 275 for 

the action faces. It facilitates understanding to note that the matrix in the equation above 
has dimension KxH . 
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30 



In a preferred embodiment of the invention, there are a total of 5 action faces, i.e., 
H = 5, and D nX9 D n29 D n39 D nA> and D n5 for n = 1,..., K denote the positions of the 

nodes of the shape mesh 275 for the action faces yawning-mouth 241, smiling-mouth 
243, kissing-mouth 245, raised-eyebrows 247, and squeezed-eyebrows 249, respectively. 
Likewise, the action vector^ has 5 components: 



^ "~ (Amy » ^ms » A MK 3 , ) 



10 A m being the amount of yawning-mouth action, A MS being the amount of smiling- 
mouth action, A MK being the amount of kissing-mouth action, A m being the amount of 
raised-eyebrows action, and being the amount of squeezed-eyebrows action. For 
example, action state z=l corresponds to yawning-mouth and has action vector 
^ = (l.O, 0.0, 0.0, 0.0, 0.0), action state i=4 corresponds to raised-eyebrows and has 

15 action vector ^ = (0.0, 0.0, 0.0, 0.0, 1.0), and action vector A = (0.5 9 0.0, 0.0, 1.0, 0.0) 
represents a half-yawning mouth and fully raised eyebrows. Still referring to FIG. 7, the 
neutral state of the face 221 is represented by the action vector 
^ = (0.0, 0.0, 0.0, 0.0, 0.0). 

20 C. Animating The 3-D Face Model (Step 140) 

Referring to FIG. 3, the method of animating the 3-D face model comprises the 
steps of (1) locally deforming the 3-D face model using the shape mesh in accordance 
with the local facial motion (Step 141); and (2) globally moving the deformed 3-D face 
25 model in accordance with the global facial motion (Step 142). In the following, a detailed 
description of these steps is given. 

- Dl . Locally Deforming The 3-D Face Model Using The Shape Mesh hi Accordance With 

The Local Facial Motion (Step 141) 



Referring to FIG. 8, given the received attachment coefficients a k9 $ k , y ki d k , 
and m k for k «l,...,i, for the nodes of the geometry mesh 273, and the moved 3-D 
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position values D f n > n = 1,..., AT, of the nodes of the shape mesh 275 in the chronological 
order, the following procedure is used to animate the adapted geometry mesh 273: 



1. Calculate the normal vectors at the nodes of the shape mesh 275: Still 
referring to FIG. 8, the normal vector n at a node of the shape mesh 275 is 
obtained by averaging the surface normals of all the triangles of the shape 
mesh 275 that have the node as one of their corners. The result of the 
averaging is normalized so that the normal vector n has unit length. 

2. Calculate the animated 3-D positions of the nodes of the geometry mesh 273: 
Still referring to FIG. 8, the location P on the triangle 281 where the node 
280 is attached to is calculated as follows. Let a(m k ), b(m k ), and c{m k ) 
denote the indices of the nodes of the triangle of the shape mesh 275 indicated 
by the index m k . Likewise, let A[ , B{ , and C{ denote the 3-D positions and 
let u k , v^, and w k denote the normal vectors of the nodes of the triangle of 
the shape mesh 275 indicated by the index m k . Hence, the following 
equations can be written: 

4' B[=D{ lmi) , and C/-^,; 

Then, the locally animated location Q k for the geometry node k is given by 
Qk = a A + Mi +7 »C/ + d k n k . 



D2. Globally Moving The Deformed 3-D Face Model In Accordance With The Global 

Facial Motion (Step 142) 
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Once the local deformations are realized on the geometry mesh 273, it is rotated 
according to the 3-D rotation matrix R f and positioned according to the 3-D 
position vector T f . Thus the globally and locally animated location S k for the 
geometry node k is given by 

S k =R f Q k +T f . 

D. Displaying The Animated 3-D Face Model (Step 150). 

The animated 3-D face model is obtained by painting the animated geometry 
mesh obtained in Step 140 with the received texture image and affine mapping data, and 
adding the received synthetic facial objects such as eyeglasses and earrings, if any. The 
animated 3-D face model of a person is displayed on the monitor 24 of the computer 
system 10 for being viewed by die same and/or other persons. 
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What is claimed is: 

1 . A method for controlling a first mesh representative of fine features of a face with a 
second mesh representative of coarse features of the face comprising the steps of: 

5 attaching the fine mesh with a first set of nodes to the coarse mesh with a second set 
of nodes; and 

deforming the fine mesh using the coarse mesh. 

2. The method of claim 1 further comprising: 

10 receiving data signals representative of movements of the nodes of the coarse mesh; 
and 

moving the coarse mesh in accordance with the received data signals in order to 
animate the fine mesh. 

15 3. A method for animating a 3-D face model comprising the steps of: 

providing a 3-D face model having a fine 3-D geometry mesh with a first set of 

j 

nodes, a coarse 3-D shape mesh with a second set of nodes, and a set of five 
attachment coefficients for each node of the fine 3-D geometry mesh, the five 
coefficients including one coefficient for triangle index, three coefficients for 
20 interpolation weights, and one coefficient for surface distance; 

attaching the fine geometry mesh to the coarse shape mesh in accordance with the 
attachment coefficients; 

receiving global and local motion values for deforming the 3-D shape mesh; 
animating the 3-D geometry mesh of the 3-D face model using the 3-D shape mesh in 
25 accordance with the received global and local motion values; and 
displaying the animated 3-D face model. 

4. The method of claim 3 wherein the step of attachment comprises: 

assigning each node of the geometry mesh to a triangle of the shape mesh indicated 
30 by the triangle index coefficient for the node; 

obtaining an attachment point for the node of the geometry mesh on the assigned 
triangle of the shape mesh, where the attachment point is defined as a weighted 
average of the coordinates of the corners of the assigned triangle using the 
interpolation weights coefficients provided for the node; 
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calculating a surface normal of the shape mesh at the attachment point as a weighted 
average of the normal vectors at the corners of the assigned triangle using the 
interpolation weights coefficients provided for the node; and 
obtaining the location of the node by adding to the attachment point a surface vector 
defined as the surface distance coefficient times the surface normal of the shape mesh 
at the attachment point. 

5. The method of claim 3 wherein the global motion values comprise 3-D orientation 
and 3-D position data of the geometry mesh and the local motion values comprise 3-D 
displacements of the nodes of the shape mesh. 

6. The method of claim 5 wherein the step of animating comprises 

moving nodes of the shape mesh in accordance with the received 3-D displacement 
data; 

calculating surface normals of the shape mesh at attachment points of all nodes of the 
geometry mesh; 

obtaining locations of the nodes of the geometry mesh by adding to their attachment 
points a surface vector defined as the surface distance coefficient times the surface 
normal of the shape mesh at respective attachment point; and 
rotating and positioning the geometry mesh in accordance with the received global 
orientation and position data. 

7. The method of claim 3 wherein the global motion values comprise 3-D orientation 
and 3-D position data of the geometry mesh and the local motion values comprise 
action vectors for the shape mesh. 

8. The method of claim 7 wherein the step of animating comprises: 

calculating 3-D displacement data for the shape mesh in accordance with the received 
action vectors; 

moving nodes of the shape mesh in accordance with the received 3-D displacement 
data; 
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calculating surface normals of the shape mesh at attachment points of all nodes of the 
geometry mesh; 

obtaining locations of the nodes of the geometry mesh by adding to their attachment 
points a surface vector defined as the surface distance coefficient times the surface 
normal of the shape mesh at respective attachment point; and 
rotating and positioning the geometry mesh in accordance with the received global 
orientation and position data. 
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FIG.l 
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FIG. 2 
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RECEIVE THE 3-D FACE MODEL 



RECEIVE THE GEOMETRY DATA SET 



RECEIVE THE TEXTURE IMAGE 



RECEIVE ANY SYNTHETIC FACIAL OBJECTS 



RECEIVE THE GLOBAL AND LOCAL FACIAL MOTION VALUES 



120 
121 



122 



123 



130 



140 



ANIMATE THE 3-D FACE MODEL 



LOCALLY DEFORM THE 3-D FACE MODEL USING THE SHAPE MESH 
IN ACCORDANCE WITH THE LOCAL FACIAL MOTION 



GLOBALLY MOVE THE DEFORMED 3-D FACE MODEL IN 
ACCORDANCE WITH THE GLOBAL FACIAL MOTION 



141 



142 



DISPLAY THE ANIMATED 3-D FACE MODEL 
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FIG. 3 




FIG. 4 



WO 02/31775 



5/8 



PCT/IB01/02735 




271 



FIG. 5 
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FIG. 6 




FIG. 7 
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FIG. 8 



